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Lie in 2018, the California Department of 


School 


California School 


DASHBOARD 


Education rolled out an updated version of the California 


Dashboard. This revision altered the look and feel of the Dashboard and added new indicators 


based on newly available data. This brief updates a 2018 analysis of the Dashboard. First, | examine 
whether the state's revisions are in line with the suggestions made in the 2018 report. | find that the 
state has made some improvements to the system, but that there is room for continued improvement. 
Second, | use data from the 2019 PACE/USC 


for the 


Rossier poll to characterize use of and support for the 


Dashboard. | find continued low use and potentially troubling equity gaps, but | also find high support 


Dashboard and a strong preference for the new Dashboard over the old. 
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Just last year, | co-wrote a technical report for the Getting Down to Facts II (GDTFII) 


project focused on the California School 
Dashboard itself, from a state-representat 


Dashboard!. That report used data from the 
ive poll of California voters, and from interviews 


with school district leaders to understand the early implementation of the Dashboard. 
The report identified several main challenges for the Dashboard, and it offered suggestions 


to address these problems: 


e First, the analysis found that very few parents were aware of or reported 
having used the Dashboard and that there were wide gaps between more and 
less educated voters in their reported use. It recommended greater effort to 


disseminate the Dashboard 


and to support users’ access to it. 


¢ Second, the analysis found that the tool was not well designed to offer 
comparisons between schools and that school ratings were highly related to 
student demographics in a way that likely unfairly punishes schools serving 
more disadvantaged students. It recommended the state allow Dashboard users 


to compare schools, espec 


ially schools with similar characteristics. 


¢ Third, the analysis found that the Dashboard’s chosen approach to measuring 
growth in student achievement (i.e., subtracting last year’s school-average 
test scores from this year’s) was poorly aligned with what research says about 
the proper ways to use growth models to isolate school effectiveness. It 
recommended the state adopt a true value-added or student growth percentile 


model. 


¢ Fourth, the analysis found that the 


Dashboard gave insufficient weight and 


priority to the performance of student groups, often relegating those data 


to back pages or allowing schools 


to have high overall ratings despite weak 


subgroup performance. It recommended prioritizing student groups in reporting 
and giving users the ability to focus their search on the main groups of interest. 


Since this report was written, two important events have happened with respect to 


the Dashboard. First, in December 20 
replacing the look and feel of the old 
January 2019, PACE and USC Rossier 
information collected on the 2018 po 


18, the state put out a new version of the 


Dashboard, 


Dashboard and updating the data. Second, in 


conducted a poll of California voters, updating the 
| used in the original report. The purpose of this 


brief is to update the September 2018 report based on these new changes and data and 
examine the extent to which the original report's recommendations still hold. 
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Changes to the Dashboard 


The state made a number of changes, substantive, technical, and stylistic, 
in their 2018 revision of the Dashboard. Substantively, new indicators were included for the 
first time. These include chronic absenteeism, college/career, and Grade 11 test scores. 
Technically, the state updated its calculation of graduation rates, added adjustments for 
school performance in schools with low test participation, switched to three-by-five grids 
(from five-by-five) for some small schools, and began including new alternative schools 
in Dashboard calculations. Stylistically, the state put all of the Dashboard data on a single 
page, changed the way that the main rankings are indicated (moving from pie pieces to 
gauges), created a new page that allows users to see the overall ratings for each individual 
school in a district, and foregrounded student group data in several places (see Figure 1). 


Figure 1. New Displays on the California School Dashboard 
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These changes are generally welcome and in line with the recommendation in the 
2018 GDTFIl Dashboard report. In particular: 


e« The inclusion of new indicators addresses some of the superintendents’ stated 
concerns that the Dashboard was mainly still focused on test scores. 


¢« The use of three-by-five grids for small schools will improve the year-to-year 
Stability of ratings for those schools. 


¢ Moving all of the information for a school to a single page likely makes the tool 
more usable and decreases the likelinood that users will miss important data. 


¢« Allowing users to see all schools in a district side-by-side increases the utility of 
the tool for comparing schools, which may be especially important for parents 
seeking to use the data to make decisions about where to enroll their children. 


Overall, these are a good first set of changes that users should appreciate. 


However, there are several recommended changes that were not made, or that 
were made in ways that were not ideal, and these might be fertile ground for future 
modifications. For example: 


¢ The comparison tool allows users to compare schools within districts, but it 
does not allow users to compare schools between districts. Given that many 
parents may have a choice set that includes schools from multiple districts 
(including both traditional public and charter schools), this could be a useful fix. 
Perhaps the most straightforward fix would be simply to allow parents to select 
as many schools as they would like in order to compare them. 


« The comparison tool allows users to sort schools on any of the indicators, 
but it does not allow users to compare multiple indicators at once, nor does 
it allow users to compare schools with similar demographics. Given the well 
established relationships between school average performance measures and 
student demographics, this approach will tend to make more affluent schools 
look more favorable and less affluent schools to look worse. The state should 
consider creating a tool that allows users to compare similar schools, as they 
used to do under the Academic Performance Index. 


« The comparison tool allows users to compare schools only on overall 
performance on each Indicator, not on student group performance. The most 
relevant comparison for a given parent, in contrast, is probably “where do 
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students who look like my child do the best?" To this end, the tool could do 
more to offer parents the opportunity to customize their experience based on 
student demographics. The state should consider modifying the tool to allow 
users to compare schools for a given subgroup. 


« The state has not changed its approach to student growth, so all of the 
concerns about the chosen approach that were described in the previous report 
remain relevant. In that previous report, we wrote the following, and every word 
of it still applies: 


The state has chosen a simplistic ‘change’ measure by merely taking 

the difference between this year’s scores and last year’s scores On each 
outcome. This approach suffers from many problems, not the least of 
which is that it does not adjust for the fact that these are different students 
being compared to one another (i.e., there are ‘cohort effects’). Especially 
for test scores, where there is a wealth of knowledge about the best ways 
to construct accountability system growth measures, there is no reason for 
the state to choose the approach it did. The state should choose a more 
appropriate growth measure, such as a two-step value-added model. 


The state is consulting with experts on ways to improve their growth measures 
(| am a part of those conversations); hopefully, after that feedback, they 
will make a choice that is better aligned with what the literature suggests. 


Clearly, there is room for continued progress in the kinds of data reported on the 
Dashboard and the ways in which they are reported. 


Awareness and Use of the Dashboard 


All of these changes were made to the Dashboard to help educators, parents, 
and the public make better use of school performance data. The 2019 PACE/USC 
Rossier poll? offers us the opportunity to take stock of public awareness and use of the 
Dashboard, as well as to examine voters’ attitudes towards the new Dashboard. 


In terms of awareness of the Dashboard, our results show little change from 2018 
to 2019. In 2018, 45 percent of voters reported having heard of the Dashboard, with 
6 percent saying they Knew a “good deal” about it and 10 percent saying they knew a 
“fair amount.” In 2019, 46 percent of voters reported having heard of the Dashboard, 
with 9 percent saying they knew a “good deal” about it and 11 percent saying they knew 
a “fair amount.” While the overall level of awareness was therefore almost completely 
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flat (and certainly within the margin of error), the increase in higher levels of awareness 
(20 percent this year versus 16 percent last year) is a bit larger. Narrowing the sample 

to just parents, we saw no increase from 2018 to 2019—66 percent of parents last year 
reported awareness versus 63 percent this year. However, as shown in Figure 2, there 
are substantial differences between parents and voters overall: In 2019, 39 percent of 
parents reported Knowing a fair amount or a lot about the Dashboard, compared to only 
20 percent of voters. 


Figure 2. Voter and Parent Awareness of the Dashboard, 2019 
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Though awareness has not increased, the proportion of voters who reported 
having been to the Dashboard website did increase from 2018 to 2019. In 2018, 12 percent 
of voters and 35 percent of parents said they had visited the Dashboard website. In 2019, 
17 percent of voters and 38 percent of parents reported having been to the website. The 
proportion reporting having visited the Dashboard more than twice also saw increases— 
from 3 percent to 6 percent for all voters and from 12 percent to 16 percent for parents. 
Again, there are substantial differences here between parents and voters: In 2019, 
358 percent of parents had visited the Dashboard, compared to only 17 percent of voters 
(see Figure 3). 
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ee 
Figure 3. Voter and Parent Use of the Dashboard, 2019 
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We also tested whether the trends we observed in the 2018 data—with more 
educated voters much more likely to be aware of or use the Dashboard—held in 2019. 
This trend did continue (see Figure 4). Looking by household income, for instance, just 
30 percent of those with incomes below $35K had heard of the Dashboard, versus 
61 percent of those with incomes above $250K. In terms of use, just 6 percent of those 
with incomes below $35K said they had visited the Dashboard, versus 30 percent of 
those with incomes above $250K. Again, the state might need to do more to reach out 
to less educated or lower income voters If the goal is to put more data in their hands 
to make decisions about where to send their children. 
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Figure 4. Awareness and Use of the Dashboard by Income, 2019 
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We also asked a variety of questions to get at the extent to which voters supported 
the Dashboard, both in general and specifically the new version. Among voters who 
reported having been to the Dashboard, support for the Dashboard was largely unchanged. 
In 2018 the margin was 51 percent with a positive view (11 percent very positive) versus 
12 percent with a negative view (2 percent very negative). In 2019 the margin was 52 percent 
with a positive view (17 percent very positive) versus 12 percent with a negative view 
(2 percent very negative). In both years, parents who had been to the Dashboard were even 
more positive (72 percent expressed a positive view (25 percent very positive) in 2018, 

71 percent expressed a positive view (32 percent very positive) in 2019). These results were 
similar when we asked users about the utility and effectiveness of the Dashboard indicators, 
with generally 55-60 percent of voters showing support and 20-30 percent showing 
opposition. These findings suggest that overall levels of support are not changing among 
users, but that some users are getting more positive in their views. 


Finally, we specifically tested whether California voters preferred the new or the 
old Dashboard. We randomly assigned half the 2019 voters to get an image of the old 
Dashboard and half to get an image of the new one. Then we asked them how positive 
or negative their impression was of what they saw. This question highlights that voters 
were much more favorable toward the new Dashboard than the old. As shown in Figure 
5, when seeing the old Dashboard, 52 percent of voters expressed a positive impression 
(10 percent very positive) and 25 percent expressed a negative impression (5 percent very 
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negative). When seeing the new Dashboard, 65 percent of voters expressed a positive 
impression (17 percent very positive) and 19 percent a negative impression (4 percent very 
negative). Parents were even more enthusiastic about the new Dashboard, with 81 percent 
expressing a positive impression (33 percent very positive) and just 11 percent a negative 
impression (3 percent very negative). 


Figure 5. Voters’ Views Toward the Old and New Dashboard 
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Overall, the poll results lead us to several conclusions: 


« There remains a problem of awareness, with few voters and even many parents, 
aware of or having used the Dashboard. 


e« There are important inequities in awareness and use of the Dashboard along 
income lines, with more affluent voters much more likely to be aware of or 
report having visited the Dashboard. 


¢« Voters and parents who have been to the Dashboard are enthusiastic about it; 
this enthusiasm is not changing much, except there may be a slight increase in 


people who are very enthusiastic. 


e Voters and parents express a strong preference for the new Dashboard over the 
old, and parents are almost uniformly positive about the new Dashboard. 
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Discussion 


The results of this analysis are promising for the state as it looks to continue 
supporting the Dashboard, but there is room for improvement. Clearly, the state has made 
some significant changes to the Dashboard, and voters and parents seem to think the new 
Dashboard is a marked improvement. At the same time, very few voters (and somewhat 
more, but far from a majority of parents) have been to the Dashboard, suggesting its 
impact on decisions will be somewhat muted. This brief has offered some additional 
suggestions for the state to consider as it continues revising the Dashboard in future years, 
though most of the recommendations of the prior report also still hold. In particular, 
the state should focus its efforts on boosting the ability of the tool to compare schools, 
choosing an appropriate growth model, and working to disseminate the tool more widely 
and encourage its use (especially among disadvantaged groups). Through these efforts to 
continuously improve the Dashboard, the state can better leverage this new tool to drive 
school improvement efforts. 
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Policy Analysis for California Education (PACE) is an independent, non-partisan research center led by faculty 
directors at Stanford University, the University of Southern California, the University of California Davis, the 
University of California Los Angeles, and the University of California Berkeley. PACE seeks to define and sustain a 


comprehensive policy reform and continuous improvement in performance at all levels 


of California's education system, from early childhood to postsecondary education and training. PACE bridges 
the gap between research and policy, working with scholars from California's leading universities and with state 
and local policymakers to increase the impact of academic research on educational policy in California. 


issues in California’ 
« Convenes seminars and briefings that make current research accessible to policy 
audiences throughout California. 
e Provides expert testimony on educational issues to legislative committees and other 


iefs, research reports, and working papers that address key policy 


S education system. 


¢ Works with local school districts and professional associations on projects aimed at 
supporting policy innovation, data use, and rigorous evaluation. 
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